The purpose of virtual reality and simulation since its beginnings has been “to create the illusion so well that you feel you are actually doing it.” While this goal is still actively being pursued, the past ten years have shown a steady evolution in virtual reality (VR) technologies. VR technology is now being used in many fields. Air traffic control simulations, architectural design, aircraft design, acoustical evaluation (sound proofing and room acoustics), computer aided design, education (virtual science laboratories, cost effective access to sophisticated laboratory environments), entertainment (a wide range of immersive games), legal/police (re-enactment of accidents and crimes), medical applications such as virtual surgery, scientific visualization (aerodynamic simulations, computational fluid dynamics), telepresence and robotics, and flight simulation are among its applications.
Until recently, the one major component lacking in VR simulations has been the sense of touch (haptics). In the pre-haptic systems, a user could reach out and touch a virtual object, but would place his/her hand right through the object, which reduces the reality effect of the environment. Haptics provide force feedback. With force feedback, a user gets the sensation of physical mass in objects presented in the virtual world composed by the computer. Haptic systems are essentially in their infancy, and improvements may still be achieved. The systems can be expensive and may be difficult to produce.
A number of virtual reality systems have been developed previously. The systems generally provide a realistic experience, but have limitations. A number of systems known to the present inventors will be discussed briefly. Example issues in prior systems include, for example, user occlusion of the graphics volume, visual acuity limitations, large mismatch in the size of graphics and haptics volumes, and unwieldy assemblies,
Rear-Projection Virtual Reality Systems
Rear-projection virtual reality (VR) systems create a virtual environment projecting stereoscopic images on screens located between the users and the projectors. Example rear-projection VR systems include the CAVE®(g and the ImmersaDesk® systems. These displays suffer from occlusion of the image by the user's hand or any interaction device located between the user's eyes and the screens. When a virtual object is located close to the user, the user can place his/her hand “behind” the virtual object. However, the hand will always look “in front” of the virtual object because the image of the virtual object is projected on the screen. This visual paradox confuses the brain and breaks the stereoscopic illusion.
Rear-projection systems displaying stereo images also can create a visually stressful condition known as the accommodation/convergence conflict. The accommodation is the muscle tension needed to change the focal length of the eye lens in order to focus at a particular depth. The convergence is the muscle tension needed to move both eyes to face the focal point. In the real world, when looking at distant objects, the convergence angle between both eyes approaches zero and the accommodation is minimum (the cornea compression muscles are relaxed). When looking at close objects, the convergence angle increases and the accommodation approaches its maximum. The brain coordinates the convergence and the accommodation. However, when looking at stereo computer-generated images, the convergence angle between eyes still varies as the 3D object moves back and forward, but the accommodation always remains the same because the distance from the eyes to the screen is fixed. When the accommodation conflicts with the convergence, the brain gets confused and many users experience headaches.
Augmented Virtual Reality Systems
Augmented reality displays are more suitable for haptics-based applications because, instead of projecting the images onto physical screens, they use half-silvered mirrors to create virtual projection planes. A user's hands, located behind the mirror, are intended to be integrated with the virtual space and therefore provide a natural interaction with the virtual space. A user can still see his/her hands without occluding the virtual objects.
The stereo effect in computer graphics displays is achieved by defining a positive, negative, or zero parallax according to the position of the virtual object with respect to the projection plane. Only when the virtual object is located on the screen (zero parallax) the accommodation/converge conflict is eliminated. Most augmented reality systems do a fair job of minimizing this conflict. Since the projection plane is not physical the user can grab virtual objects with his/her hands nearby, or even exactly at, the virtual projection plane.
However, the conflict can still arise for a number of reasons. If head tracking is not used or fails to accommodate a sufficient range of head tracking, then the conflict arises. In systems with head tracking, if the graphics recalculation is slow then the conflict arises. In systems lacking head tracking, the conflict arises with any user movement. Systems that fail to permit an adequate range of movement tracking can cause the conflict to arise, as well, as can systems that do not properly position a user with respect to the system. The latter problem is especially prevalent in systems requiring a user to stand.
PARIS™
PARIS™ is a projection-based augmented reality system developed by researchers at the University of Illinois at Chicago that uses two mirrors to fold the optics and a translucent black rear-projection screen illuminated by a Christie Mirage 2000 stereo DLP projector. A user stands and looks through an inclined half-silvered mirror that reflects an image projected onto a horizontal screen located above the user's head. A haptics volume is defined below the inclined half-silvered mirror, and a user can reach his/her hands into the haptics volume.
The horizontal screen is positioned outside of an average sized user's field of view, with the intention that only the reflected image on the half-silvered mirror is viewable by the user when the user is looking at the virtual projection plane. Because the half-silvered mirror is translucent, the brightness of the image projected on the horizontal screen is higher than the brightness of the image reflected by the mirror. If the user is positioned such that the image on the horizontal screen enters the field of view, the user can be easily distracted by the horizontal screen.
An issue in haptic augmented reality systems is maintaining collocation of the graphical representation and the haptic feedback of the virtual object. To maintain certain realistic eye-hand coordination, a user has to see and touch the same 3D point in the virtual environment. In the PARIS™ system, collocation is enhanced by a head and hand tracking system handled by a dedicated networked “tracking” computer. Head position and orientation is continuously sent to a separate “rendering” PC over a network to display a viewer-centered perspective. In the PARIS™ system, the tracking PC uses a pcBIRD, from Ascension Technologies Corp. for head and hand tracking.
The PARIS™ system uses a large screen (58″×47″), and provides 120° of horizontal field of view. The wide field of view provides a high degree of immersion. The maximum projector resolution is 1280×1024@108 Hz. With the large screen used in the PARIS™ system, the pixel density (defined as the ratio resolution/size) is 22 pixels per inch (ppi), which is too low to distinguish small details.
Visual acuity is a measurement of a person's vision. Perfect visual acuity is 20/20. Visual acuity for displays can be calculated as 20/(FOV*1200/resolution)(FOV=field of view). In PARIS™, this is 20/(120°*1200/1280 pixels)=20/112.5. Poor visual acuity makes reading text associated with a display very uncomfortable. It can lead to visual fatigue, headaches, dizziness, etc.
The PARIS™ system uses a Sensable Technologies' PHANTOM® Desktop™ haptic device, which presents a haptics workspace volume that approximates a six-inch cube. The graphics workspace volume exceeds the haptics volume considerably. This mismatch of haptics and graphics volume results in only a small portion of the virtual space to be touched with the haptic device. Additionally, with the mismatched volumes only a small number of pixels are used to display the collocated objects.
The PARIS™ system's use of an expensive stereo projector, and its large screen and half-silvered mirror, require use of a cumbersome support assembly. This support assembly and the system as a whole do not lend themselves to ready pre-assembly, shipping, or deployment.
Reachin Display
The Reachin display is a low-cost CRT-based augmented reality system. A small desktop-sized frame holds a CRT above a small half-silvered mirror that is slightly smaller in size than the 17″ CRT. The CRT monitor has a resolution of 1280×720@120 Hz. Since the CRT screen is 17 inches diagonal, the pixel density is higher than that of PARIS™: approximately 75 ppi. With a horizontal FOV of 350, the visual acuity is 20/(350*1200/1280)=20/32.81, resulting in a better perception of small details. However, the image reflected on the mirror is horizontally inverted; therefore, the Reachin display cannot be used for application development. To overcome this drawback, it is necessary to use the proprietary Reachin applications programming interface (API) to display properly inverted text on virtual buttons and menus along with the virtual scene.
The Reachin display lacks head tracking. The graphics/haptics collocation is only achieved at a particular sweet spot, and totally broken as soon as the user moves his/her head to the left or right looking at the virtual scene from a different angle. In addition, the image reflected on the mirror gets out of the frame because the mirror is so small. The position of the CRT is also in the field of view of the user, which is very distracting.
2.3 SenseGraphics 3D-MIW
SenseGraphics is a portable auto-stereoscopic augmented reality display ideal for on-the-road demonstrations. A Sharp Actius RD3D laptop is used to display 3D images without requiring the wearing of stereo goggles. It is relatively inexpensive and very compact. The laptop is mounted such that its display is generally parallel to and vertically above a like-sized half-silvered mirror. Like most auto-stereoscopic displays, the resolution in 3D mode is too low for detailed imagery, as each eye sees only 512×768 pixels. The pixel density is less than 58 ppi. With a FOV of 35°, the visual acuity is 20/(35°*1200/512 pixels)=20/82.03. Like the Reachin display, the haptics/graphics collocation is poor because it assumes that a user's perspective is from a single fixed location. The laptop display has its lowest point near the user and is inclined away toward the back of the system. This is effective in making sure that the display of the laptop is outside the view of a user. However, there is a short distance between the laptop display and the mirror. This makes the user's vertical field of view too narrow to be comfortable. Also, as in the Reachin display, the image is inverted, so it is not well-suited for application development. Recently Sensegraphics has introduced 3D-LIW, which has a wider mirror; however the other limitations still exist.